The present invention relates to a technique such as crowdsourcing for processing a task by people using a computer system, and more specifically, it relates to a technique for ensuring accuracy of a task operation.
In recent years, as computer networks and web services have been developed, a business form called crowdsourcing, which is outsourced to a large number of unspecified individuals, has increased.
Crowdsourcing is a technique for performing a task in a collective intelligence way by distributing a task, such as addition of captions to images, transcribing speech information, or translation, that is technically difficult to be automated by a computer and that can be subdivided, to many people as micro-tasks.
The use of crowdsourcing enables a service that could not be achieved by automation by a computer, such as “service that returns description of objects appearing on any sent photograph.” Thus crowdsourcing is used in various fields.
A problem in that case is that because of a characteristic of participation of many workers vary in quality and the quality cannot be ensured. For example, with the existing techniques, it is difficult to set a target of accuracy at 99%.
To ensure operation accuracy in crowdsourcing, techniques for integrating results of a plurality of participants have been proposed. Some examples of such related techniques are described below.
One known example is a majority scheme that is a technique for making a decision by means of a majority decision of answers of participants. However, because the majority scheme cannot consider the skills of participants, it has the drawback of low efficiency.
Another known example is a verification scheme using verification of an expert. The verification scheme ensures accuracy in a sense that verification based on expertise of an expert is conducted, but if a finite number of experts verify all tasks, the verifications cause a bottleneck in processing.
Still another known example is an additional majority scheme. For this technique, two persons first process a task, and if results are different, a majority decision is made. This technique has small expandability, and is required to combine with the verification scheme. To improve the above issues, techniques described in the papers listed below have been proposed.
P. Welinder, S. Branson, S. Belongie, and P. Perona, “The multidimensional wisdom of crowds,” In Proceedings of the 24th Annual Conference on Neural Information Processing Systems (NIPS), 2010, pp. 2424-2432 discloses improving the accuracy in majority decision by estimating the skills of workers in crowdsourcing and integrating opinions of the workers on the basis of the estimated skills. This technique, however, cannot ensure target accuracy.
Pinar Donmez, Jaime G. Carbonell, and Jeff Schneider, “Efficiently learning the accuracy of labeling sources for selective sampling,” In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD '09), ACM, New York, N.Y., USA, 2009, pp. 259-268 discloses finding reliable workers as fast as possible from operation histories of workers in crowdsourcing, causing only the reliable workers to perform a task, and making a majority decision. This technique, however, is based on repeated operations by only reliable workers. Thus it is difficult to deal with a situation where less reliable workers are also contained to ensure a sufficient number of workers.